Intelligent Control of Closed-Loop Sedation in Simulated ICU Patients

The intensive care unit is a challenging environment to both patient and caregiver. Continued shortages in staffing, principally in nursing, increase risk to patient and healthcare workers. To evaluate the use of intelligent systems in the improvement of patient care, an agent was developed to regulate ICU patient sedation. A temporal differencing form of reinforcement learning was used to train the agent in the administration of intravenous propofol in simulated ICU patients. The agent utilized the well-studied Marsh-Schnider pharmacokinetic model to estimate the distribution of drug within the patient. A pharmacodynamic model then estimated drug effect. A processed form of electroencephalogram, the bispectral index, served as the system control variable. The agent demonstrated satisfactory control of the simulated patient’s consciousness level in static and dynamic setpoint conditions. The agent demonstrated superior stability and responsiveness when compared to a well-tuned PID controller, the control method of choice in closed-loop sedation control literature. Introduction The Intensive Care Unit (ICU)1 represents a challenging environment to patient and staff alike. ICU patients may experience high anxiety levels from the general environment, and effective sedation2 is necessary to soothe the patient and to obliterate asynchronous breathing or movement that might interfere with adequate oxygenation (Kowalski & Rayfield 1999). In recent years, a shortage of ICU nurses has resulted in patient mortality (Aiken et al. 2002; Lasalandra 2001) among other negative outcomes. Norrie (1997) observed that the ICU nurse’s greatest time expense was “direct nursing care,” including the administration of intravenous sedating drugs. It is thus reasonable to conclude that automating some aspects of the intensive care environment (like patient sedation) can positively impact overall ICU patient care. Reinforcement learning (RL) represents a relatively new framework for constructing and applying intelligent agents. Copyright © 2004, American Association for Artificial Intelligence (www.aaai.org). All rights reserved. 1“Intensive Care Unit” is a general label for several critical care environments including the Medical, Surgical, Cardiac, and Burn ICUs. 2Sedation is a drug-induced depression of consciousness (Task Force on Sedation and Analgesia by Non-Anesthesiologists 1996). RL merges ideas from stochastic approximation and optimal control theory with the traditional concept of the intelligent agent. Much of the current research in reinforcement learning has been dedicated to autonomous robotic applications (Russell & Norvig 1995; Sutton & Barto 1998). RL has demonstrated favorable results in the associated problem domains; however, the extent of RL’s aptitude for other specialized planning tasks remains incompletely explored. Several encouraging works exist: Guallapalli demonstrated that RL could successfully perform closed-loop control of the benchmark peg-in-hole task (Gullapalli 1993), and Hu applied some of the founding principles of reinforcement learning to anesthesia control with favorable results (Hu, Lovejoy, & Shafer 1994).